{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Redis列表实现一次pop 弹出多条数据\n",
    "\n",
    "![](https://kingname-1257411235.cos.ap-chengdu.myqcloud.com/2019-03-03-16-52-34.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 连接 Redis\n",
    "\n",
    "import redis\n",
    "client = redis.Redis(host='122.51.39.219', port=6379, password='leftright123')\n",
    "\n",
    "# 注意：\n",
    "# 这个 Redis 环境仅作为练习之用，每小时会清空一次，请勿存放重要数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 准备数据\n",
    "\n",
    "client.lpush('test_batch_pop', *list(range(10000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "循环读取10000条数据，使用 lpop 耗时：112.04084920883179\n"
     ]
    }
   ],
   "source": [
    "# 一条一条读取，非常耗时\n",
    "import time\n",
    "\n",
    "\n",
    "start = time.time()\n",
    "while True:\n",
    "    data = client.lpop('test_batch_pop')\n",
    "    if not data:\n",
    "        break\n",
    "end = time.time()\n",
    "\n",
    "delta = end - start\n",
    "print(f'循环读取10000条数据，使用 lpop 耗时：{delta}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 为什么使用`lpop`读取10000条数据这么慢？\n",
    "\n",
    "因为`lpop`每次只弹出1条数据，每次弹出数据都要连接 Redis 。大量时间浪费在了网络传输上面。\n",
    "\n",
    "## 如何实现批量弹出多条数据，并在同一次网络请求中返回？\n",
    "\n",
    "先使用 `lrange` 获取数据，再使用`ltrim`删除被获取的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[b'9999',\n",
       " b'9998',\n",
       " b'9997',\n",
       " b'9996',\n",
       " b'9995',\n",
       " b'9994',\n",
       " b'9993',\n",
       " b'9992',\n",
       " b'9991',\n",
       " b'9990']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 复习一下 lrange 的用法\n",
    "\n",
    "datas = client.lrange('test_batch_pop', 0, 9)  # 读取前10条数据\n",
    "datas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 学习一下 ltrim 的用法\n",
    "\n",
    "client.ltrim('test_batch_pop', 10, -1)  # 删除前10条数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "现在列表里面还剩9990条数据\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[b'9989',\n",
       " b'9988',\n",
       " b'9987',\n",
       " b'9986',\n",
       " b'9985',\n",
       " b'9984',\n",
       " b'9983',\n",
       " b'9982',\n",
       " b'9981',\n",
       " b'9980']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 验证一下数据是否被成功删除\n",
    "\n",
    "length = client.llen('test_batch_pop')\n",
    "print(f'现在列表里面还剩{length}条数据')\n",
    "datas = client.lrange('test_batch_pop', 0, 9)  # 读取前10条数据\n",
    "datas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[b'9989',\n",
       " b'9988',\n",
       " b'9987',\n",
       " b'9986',\n",
       " b'9985',\n",
       " b'9984',\n",
       " b'9983',\n",
       " b'9982',\n",
       " b'9981',\n",
       " b'9980']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 一种看起来正确的做法\n",
    "\n",
    "def batch_pop_fake(key, n):\n",
    "    datas = client.lrange(key, 0, n - 1)\n",
    "    client.ltrim(key, n, -1)\n",
    "    return datas\n",
    "\n",
    "batch_pop_fake('test_batch_pop', 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[b'9979',\n",
       " b'9978',\n",
       " b'9977',\n",
       " b'9976',\n",
       " b'9975',\n",
       " b'9974',\n",
       " b'9973',\n",
       " b'9972',\n",
       " b'9971',\n",
       " b'9970']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.lrange('test_batch_pop', 0, 9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 这种写法用什么问题\n",
    "\n",
    "在多个进程同时使用 batch_pop_fake 函数的时候，由于执行 lrange 与 ltrim 是在两条语句中，因此实际上会分成2个网络请求。那么当 A 进程\n",
    "刚刚执行完lrange，还没有来得及执行 ltrim 时，B 进程刚好过来执行 lrange，那么 AB 两个进程就会获得相同的数据。\n",
    "\n",
    "等 B 进程获取完成数据以后，A 进程的 ltrim 刚刚抵达，此时Redis 会删除前 n 条数据，然后 B 进程的 ltrim 也到了，再删除前 n 条数据。那么最终导致的结果就是，AB 两个进程同时拿到前 n 条数据，但是却有2n 条数据被删除。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用 pipeline 打包多个命令到一个请求中\n",
    "\n",
    "pipeline 的使用方法如下：\n",
    "\n",
    "```python\n",
    "import redis\n",
    "\n",
    "client = redis.Redis()\n",
    "pipe = client.pipeline()\n",
    "pipe.lrange('key', 0, n - 1)\n",
    "pipe.ltrim('key', n, -1)\n",
    "result = pipe.execute()\n",
    "```\n",
    "\n",
    "pipe.execute()返回一个列表，这个列表每一项按顺序对应每一个命令的执行结果。在上面的例子中，result 是一个有两项的列表，第一项对应 lrange 的返回结果，第二项为 True，表示 ltrim 执行成功。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 真正可用的批量弹出数据函数\n",
    "\n",
    "def batch_pop_real(key, n):\n",
    "    pipe = client.pipeline()\n",
    "    pipe.lrange(key, 0, n - 1)\n",
    "    pipe.ltrim(key, n, -1)\n",
    "    result = pipe.execute()\n",
    "    return result[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 清空列表并重新添加10000条数据\n",
    "client.delete('test_batch_pop')\n",
    "client.lpush('test_batch_pop', *list(range(10000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "批量弹出10000条数据，耗时：0.18534111976623535\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "while True:\n",
    "    datas = batch_pop_real('test_batch_pop', 1000)\n",
    "    if not datas:\n",
    "        break\n",
    "    for data in datas:\n",
    "        pass\n",
    "end = time.time()\n",
    "print(f'批量弹出10000条数据，耗时：{end - start}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.llen('test_batch_pop')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![读者交流QQ群](https://kingname-1257411235.cos.ap-chengdu.myqcloud.com/2019-02-16-09-59-56.png)\n",
    "![](https://kingname-1257411235.cos.ap-chengdu.myqcloud.com/640.gif)\n",
    "![](https://kingname-1257411235.cos.ap-chengdu.myqcloud.com/2019-03-03-20-47-47.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
